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ABSTRACT 

This report attempts to profile students who 
participated in a computer-assisted language learning (CALL) research 
and development project. The researchers hoped to find some pattern 
or common denominator within each group i.e., the successes, the 
failures, and the dropouts, that would identify group membership and 
distinguish one group from another in order to further understand the 
characteristics cf language learners and to be in a position to 
rationally design alternative methods of computer-assisted language 
learning for failures and dropouts. To accomplish this object5ve, 
three groups of factors were examined, namely, individual 
characteristics, strategy use, and course information, to try to 
account for students' differential performance. The subjects for the 
study were 158 students enrolled in freshman composition classes 
during the spring semester, 1990, at Northern Arizona University in 
Flagstaff. Participants undertook four CALL computerized reading and 
note-taking lessons; two achievement tests were part of each lesson, 
so that altogether there were eight measures of achievement. After 
finishing all four lessons, students turned in four types of data 
from each of the lessons — notes, recall, recognition, and attitude. 
Scores were then computed for the recall and recognition parts and 
these scores formed the basis of the grouping criteria. Methods used 
to assess the three types of factors that might have contributed to 
group membership are explained next, and finally the Discriminant 
Function Analysis out of which a profile for the subjects in each 
group emerged, is discussed. It is concluded that with more 
information about what students are like; i.e., what kind of profile 
describes a successful or unsuccessful student, aids can be' 
incorporated into computer assisted lessons that will help students 
who experience difficulty to overcome their problems rather than 
leaving them with no alternative but to be forced into the mold of 
the "ideaT* successful student. (KM) 
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Successes, Failures, and Dropouts in Computer-Assisted Language 

Lessons 

Joan Jamieson, Leslie Norfleet, and Nora Berbisada 
Northern Arizona University 



Introduction 

Successes, failures, and dropouts in computer-assisted 
language learning (CALL) lessons attempts to profile students who 
participated in a CALL research and development project. It was 
our hope to find some pattern, some common denominator, within 
each group--the successes, the failures, and the dropouts-"-that 
would identify group membership and distinguish one group from 
another. If we could do this, we felt we would further our 
understanding of the characteristics of language learners and 
would be in a position to rationally design alternative methods 
of computer-assisted language learning for failures and dropouts. 

To accomplish this objective, three groups of factors were 
examined--individual characteristics, strategies and course 
inf ormation--to try to account for students' differential 
performance. In order to establish the setting for our results, 
first the students who participated in the study will be briefly 
described. Then, the CALL lessons which were used v/ill be 
explained. Next, the factors that we thought might contribute to 
group membership will be explained. Then, the analysis that we 



used will be described and our results will be presented. 
Finally, we will discuss our findings. 

Subjects 

The subjects for this study were students enrolled in 
freshman composition classes during the Spring semester, 1990, at 
Northern Arizona University in Flagstaff. A total of 258 students 
volunteered to do four computerized reading and note-taking 
lessons (described below). Two achievement tests v/ere part of 
each lesson, so that all together there were eight measures of 
achievement. Forty-one students scored above the 70th percentile 
on all eight different achievement measures; twenty-seven 
students scored below the 30th percentile on the same eight 
measures; ninety students dropped out of the project after having 
agreed to participate. These three groups of students, totalling 
158, comprised our successes, failures, and dropouts. 

Of the forty-one "successful" students, there were fifteen 
males and twenty-six females; the median age was nineteen. Of the 
twenty-seven "failures," eight were male and nineteen were 
female; the median age was twenty. Among the ninety "dropouts," 
forty-one v;ere male and forty-nine were female; nineteen was the 
median age. 



Computer Lessons 

We designed four computer lessons for our study. Research 
done by Stahl, Jacobson, Davis and Davis (1989), has shown that 
prior knowledge, or background knowledge, has an impact on how 
well the students recall information at a later date. We 
therefore insured that each of these lessons dealt with a 
different topic in order to account for the students' schema. 
These topics were selected from typical readings that college 
freshmen will have for class, as we thought that the readings 
should be appropriate for the particular level of our subjects. 
Although including examples of different types of genres has 
proven to be beneficial to students (Smith, 1991), we decided to 
limit our lessons to one genre, that of expository prose. This 
was to control for the effects that different genres might have 
on the study. The topics of music appreciation, electric 
conductivity, minerals, and Aztec civilization were thus chosen. 

Although these lessons covered four different topics, they 
had a uniform method of presentation, based in part on a study by 
Kiewra and Frank (1986). Each lesson was divided into four parts: 
a) reading and note-taking, b) recall, c) recognition quiz, and 
d) attitude questionnaire. Each, lesson was done by a student in 
two consecutive sessions: reading and note-taking (part a) was 
done during the first session, while the other three parts were 
done at least one day later during the second session. 



The lessons were presented on a VAX computer system and were 
programmed in Digital's Authoring Language (DAL) and "C." After 
the student saw the title and directions, he was then presented 
with the first paragraph of the reading for the reading and 
note-taking task. In determining the screen display, certain 
recommendations for programming were followed. First, the student 
needs to be able to find the "Help" prompt in the same place. 
Second, paragraphs were kept coherent, not breaking up ideas that 
naturally need to be viewed together. Finally, the setup v/as such 
that no part of the screen overshadowed the content portion, 
since this is the most important part of the lesson (England, 
1989). The screen, then, was divided into three windows: the top 
windov; contained one paragraph at a time from the reading; the 
middle window contained important key words; the third window was 
an empty space where the student's typed-in notes would appear. 
When the student completed the reading, he was instructed to quit 
working and return to the computer lab another day. 

The second session began with recall. A subsection of the 
recall part was the review of notes. In this particular section, 
the student was shown all the notes that he had prieviously 
written during the first session. There was no limit as to the 
time it took for the student to review. After reviewing, the 
student then went on to type everything he could remember 
(recall) about the lesson. 

After the recall came a recognition quiz. The questions in 
this section were designed to te^t the main ideas and important 
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subpoints (generally factual knowledge) " of the reading* A 
multiple choice format with only one correct answer and three 
foils was used* There were twenty questions* 

The last part of this session was the attitude questionnaire 
which consisted of eleven questions about the student^ 's 
background knowledge and opinions about the lesson* Ba^^d on a 
Likert scale, a student read a statement and then typed l^to 5 
according to the extent of his agreement or disagreement* 

After finishing all four lessons, the student gave us 16 '\ 
sets of data---notes , recall, recognition, and attitude---f our 
types of data from each of the four lessons* From these, scores 
were computed for the recall and recognition parts of all four 
lessons; these scores formed the basis of our grouping criteria* 
Those whose total scores on all eight measures (four recall and 
four recognition) were at or above the 70th percentile were 
placed in .he success group* Those whose total scores were at or 
below the 30th percentile were placed in the failure group. 
Students who did not do the lessons after having agreed to 
participate in the study were placed in the dropout group* 

The Factors 

Three types of factors that might account for group 
membership of the successes, failures, and dropouts were 
investigated* These main areas were individual characteristics of 
the learner, strategy use, and course information* Each of these 
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potential discriminators will be discussed in turn prior to an 
explanation of the analysis and the description of results • 

The first potential factor had to do with individual 
characteristics of the students in our study. Six different 
characteristics were of interest. The first few, sex (SEX), age 
(AGE), first language (LGl), second language (LG2), and the 
amount of months between high school and college (TMAWAY) were 
recorded when the students initially agreed to participate in the 
study. SEX, LGl, LG2, and TMAWAY were treated as dichotomous 
data. 

Finally, we had the students take a cognitive style test, 
specifically the Group Embedded Figures Test, which is a measure 
of Field Independence/Dependence (FI/D) (Witkin, Oltman, Raskin 
and Karp, 1971), FI/D is defined as "the extent to which a person 
perceives part of a field as distinct from the surrounding field 
as a vrhole," Students scoring high on this standardized test are 
characterized as analytical, imposing their own structure on 
material to be learned. People who score low on this measure, on 
the other hand, are characterized as holistic learners, 
preferring to learn material that is already structured, FI/D was 
treated as a continuous variable (FISCORE) whose values ranged 
from. 0-18, 

These individual characteristics, then, that might possibly 
add to the profile of students labelled as successes, failures, 
and dropouts included the variables of sex^^age, first language, 
second language, time away from school, and cognitive style. 
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A second set of factors that were of interest to us was 
strategy use by our lesson participants ♦ In the classroom, when 
the test for FI/D was given, students also completed a 
questionnaire. This questionnaire was an adapted version of the 
Strategy Inventory for Language Learning (SILL), (Oxford, 1990), 
The SILL'S questions distinguish between two main types of 
strategies: direct, strategies and indirect strategies. 

The first group, the direct strategies, can be defined as 
those strategies that involve direct contact with language use. 
The direct strategies include memory, cognitive, and compensatory 
strategies. The first of these subgroups, memory strategies, are 
those that enable the student to store and access information, 
utilizing techniques that have been around for a long time. 
Examples of memory strategies include grouping, making 
associations, and placing new words in context. In our analysis, 
we named this variable MEr4AV. 

The second direct strategy, use of cognitive techniques, 
involves the manipulation or transformation of language by the 
learner. Examples of cognitive strategies include repeating, 
practicing sounds and writing, making use of formulas and 
patterns, recombining familiar items in new ways, skimming and 
scanning, taking notes, summarizing, and analyzing material. 
These strategies are the ones that are most familiar to 
students, and as a result, are the most popular. In our analysis 
this variable was called COGAV. 
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The final direct strategy involves compensation • 
Compensatory strategies are those that enable students to make up 
for where they are limited in their knowledge of a language* 
Compensation techniques involve using all the clues possible to 
understand language^ whether it is the main idea in a passage or 
the gist of a conversation. The use of circumlocution^ synonyms, 
and gestures to get an idea across exemplifies compensation 
strategies. This variable was COMPAV in our analysis. 

The other main category of strategies involves those termed 
indirect strategies. As their name indicates, these are 
strategies that do not entail direct contact with language, but 
rather address strategies that indirectly involve language. The 
first of these strategies is metacognitive strategies which can 
be defined as those techniques that allow a student to manage his 
own cognition. and coordinate how learning occurs. Examples 
include overviewing material, linking new material with what is 
already known, consciously deciding to pay attention, and 
arranging a schedule for learning. This variable was called 
METAV. 

The- second indirect strategy involves all of the affective 
strategies. These techniques enable one to keep a handle on his 
emotions, motivations, and attitudes. Examples include lowering 
anxiety levels when possible, using positive statements, taking 
wise risks, rewarding oneself for doing well, and noting when 
stress is too great. We called this variable AFFAV. 
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The last type of indirect strategy is the use of social 
strategies. These techniques assist learners because they involve 
interaction with others. Asking questions for clarification, 
asking for verification of what's been said, and asking for 
corrections exemplify social strategies that may be used in 
learning. SOCAV was the name of this strategy. 

A final factor that we thought might account for group 
membership was the course information J-hat was obtained for each 
student. The information tfeat we were most interested in included 
semester grade point average (SEMGPA), cumulative GPA (CUMGPA), 
total number of credits taken for the semester (USEM), and total 
number of cumulative credits taken (TUCUM). 

Analysis 

Once all of these data were obtained for each student, along 
with the Information stored from the computer lessons themselves, 
we had to ask oursel-.v^s how to best analyze this statistically. 
We were interested in student profiles. In order, then, to 
investigate the groups that had been identified (that is, the 
successes, the failures, and the dropouts) from the groups of 
variables just described. Discriminant Function Analysis was 
chosen as the multivariate statistic. 

The major purpose of Discriminant Function Analysis 
according to Tabachnick and Fidell (1989) is "to predict group 
membership from a set of predictors" (p. 505). This statistical 
procedure enables researchers to predict whether a set of 
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variables can reliably predict group membership* This is 
basically what we wanted to do* We wanted to find a dimension 
among our set of variables which would enable us to define a 
student profile for each of the groups that we created. However, 
.one of the assumptions of discriminant function analysis is that 
the number of variables multiplied by 5. should not exceed the 
total sample size (Tabachnick & Fidell, 1989). Since our total 
sample size is 158 and we have a total of 17 variables, we 
decided to have the least probability of error by running the 
discriminant funtion in four analyses. The first three analyses 
are based on the three categories of factors v/hich we thought 
might influence group membership (individual characteristics, 
strategy use, and course information). Then, based on the results 
of these analyses, we selected the variables which discriminated 
the most and ran the final analysis which would enable us to 
define the student profiles. 

Our total sample of 158 cases was divided into three grouper: 
group 1 was labeled the success group and had 41 cases; group 2 
was the failure group and had 27 cases; group 3, the dropout 
group, had 90 cases. Discriminant Function Analysis is robust to 
uneven sample sizes because it is typically a one way analysis 
(J. D. Petersen, personal coirimunication, February, 1992), thus 
our uneven sample sizes did not create a special problem. 
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Results 

For the first analysis, we investigated the combination of 
individual characteristics in terms of six variables, SEX, AGE, 
LGl, LG2, TMAWAY, and FISCORE. Table 1 shows the summary of th.. 
discriminant functions. Function 1, or the first combination of a 
set of predictors causing variance among the groups, accounted 
for 87% of the total variance while the second function accounted 
for only 13%. The canonical correlation, which is the correlation 
between the grouping variables and the sum of the discriminant 
scores on the set of predictors is .30, much -higher than the 
second function. The Wilks' Lambda, the main statistical 
procedure used in the stepwise analysis, is ,896 which is lower 
than the second f unction--the lower the value of Wilks' Lambda 
the more significant the function. Thus, you will see that the 
first function is significant at ,002 but the second function is 
not significant at all. 

Insert Table 1 about here, 
***************************************** 

The next step investigated the question, "We know that the 
first function is significant, but among all the variables, which 
accounted for the most variation?" The canonical discriminant 
function coefficient tells us that FISCORE has the highest beta 
weight, with a loading of .99. In Discriminant Function Analysis, 
only those variables with a loading of .3 or higher are 
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considered to be significant. Therefore, since none among all of 
the other variabxes showed a loading of ,3 or more, this means 
that FISCORE is the sole variable responsible for variation among 
the groups. Since Function 2 is not significant, the beta weights 
and loadings were not included in Table !♦ 

The second analysis was based on strategy use and as stated 
above, six variables were included. Function 1 accounted for 80% 
of the total variation v/ith a canonical correlation of .365, a 
Wilks* Lambda of .833 and a significance level of .005. The 
second function is not significant. Looking at the beta weights 
and loadings, we see that though COGAV (cognitive strategies) has 
a beta weight of 1.116, its loading is only .34. No one variable 
among this set accounted for a huge part of the variation, thus 
we might be able to include AFFAV (affective strategies) with a 
loading 'of .28 as also an important variable. Since there is no 
one variable accounting for a very high loading, then, a loading 
of .28 might be considered sufficient (J. D. Petersen, persona' 
communication, February, 1992). 

■k-k-kici^ic-kificicicic'kiricic-kiriciciciricicicicic-kicicicicicic'y'k-k 

Insert Table 2 about here. 

★ Vc**** ************** 

The third analysis (see Table 3) dealt with the course 
information variables. The summary table shows that function 1 
v/as significant as it accounted for 93% of the variance. Looking 
at the beta v/eights and loadings, cumulative GPA had the highest 
values. Moreover, all the rest of the variables (semester GPA, 



total units in that semester, and total cumulative units) were 
all significant with values over .3. 

Insert Table 3 about here. 

•k-k-k-k-k-k-k-k k^-kk'kii-k-k-k-k-k-k-k'kk-k'k'k'k'k'k'k'k'k'k'k'k'k-kk 

We considered the fourth analysis the most crucial since it 
dealt with all the variables considered important based on their 
loadings for significant functions • Thus we ran an analysis 
including the Field Independence/Dependence score v/hich was the 
most important predictor for Analysis 1, the cognitive and 
affective strategies which were the most important predictors for 
Analysis 2, and the number of units taken in the semester, total 
number of units taken, semester GPA and cumulative GPA, which 
were all significant in Analysis 3, 

Table 4 shows that two functions are significant ♦ The 
loadings on the first function shows that the combination of all 
four course information variables are important predictors for 
group membership. In the second function, the Field 
Independence/Dependence score is the variable which accounted for 
most of the variation at ,85. 

Insert Table 4 about here. 

kk-k-kkk'k-kkk-k-k-k^k-k-k-k-k-k-k'k-k-k-kkicic'k-k'k'k'k-k-k-k-k-k-k-k-k-k'k 

To check the validity of the functions, we ran a plotting of 
the group means (known as the group centroids) based on the two 
functions. In Figure 1, the x axis represents the first function 
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and the y axis represents the second function. Looking at 
function 1 (course information), you will notice that group 2 
(Failures) and group 3 (Dropouts) are distinctly separated from 
group 1 (Successes) • For function 2 (cognitive style) /FI/D) , 
notice that group 2 (Failures) is separated from group 1 and 
group 3. So you see, different separations of groups were 
accounted for by different combinations of variables. 

Insert Figure 1 about here. 

Discussion 

From discriminant analysis, we were thus able to define a 
student profile for the subjects in each group. The successful 
student was one who had a high semester and cumulative GPA, took 
a ] igh number of units both during the semester and cumulatively, 
and had a high Field Independence/Dependence score. On the other 
hand, a student who belonged to the Failure group was one who had 
a low semester and cumulative GPA, had taken fewer semester and 
cumulative units, and had a low Field Independence/Dependence 
score. There v/as a complexity ip the student profile of the 
Dropout group because though he had a low semester and cumulative 
GPA and few semester and cumulative units, his Field 
Independence/Dependence score was only a little bit lower that 
the Success group average, but much higher than that of the 
Failure group. 
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With these analyses in mind, then, how can CALL lessons 
address differences found among our successes, failures, and 
dropouts? Studies have shown that students in general are more 
active participants when participating in CALL lessons (Rauch, 
1983), The question then, is not whether they should engage in 
CALL lessons, but rather how we can tailor them for the groups 
outlined* 

With advances in programming, the CALL lessons can be 
designed to vary lesson presentation as the students participate 
in the lessons. Or, with such analyses like the ones done here, 
we might consider predicting "at risk" students. These students 
might then be monitored and/or reinforced while working on the 
lessons* As we are reminded by Chapelle and Mizuno (1989), we 
need to vjatch what students are actually doing while they are 
taking the lessons, so that we can better address their needs. 

Computers are an ever increasing resource that have great 
promise for the future. To best utilize them to their potential, 
however, we must not forget that they are to assist us and that 
the adaptation need not be on the part of our CALL participants, 
but rather the adaptation can be on the part of the computer. 
Most of us have experienced not working well on the computer. 
This kind of frustration is not often addressed, since the 
successful students are the ones who finish the lessons, 
activities, etc., when working on the computer. But what about 
the student who has difficulty? What can we do as programmers 
and instructors to change his experience into one of success? In 
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the educational system as it is today, these kinds of students do 
get lost in the shuffle, yet computerized individual instruction 
may be one way that these students can be successful, if the 
computer's potential is not overlooked. Why not program strategy 
training right into the machine, so that if a student keeps 
getting a wrong answer, some helpful hints are provided? What 
about an automatic highlighting of main ideas? In the work done 
by England (1989), students had to edit incorrect portions of 
text by highlighting the mistakes and correcting them. A study 
such as this can provide us with a great deal of information 
about what students do with text. By incorporating such 
techniques then, little by little, aids can be removed, as the 
failure arxd/or dropout begins to experience more and more 
success . 

The advantages to individualizing instruction are numerous 
and given the importance of individual differences and 
uniqueness, it is no longer necessary to use one program for all 
students. With more information about what students are like, 
that is, what kind of profile describes a successful or 
unsuccessful student, wc can better assist students rather that 
forcing them into a mold of what was previously considered the 
ideal successful student. In the future, ideally, all students 
will have the chance to do well, at least where CALL lessons are 
concerned. 
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Table 1, Discriminant Function Analysis of Individual 
Characteristics 



Suironary Table of Canonical Discriminant Functions 



PERCENT OF CANONICAL WILKS ' 

FUNCTION VARIANCE CORRELATION LAMBDA SIGNIFICANCE 



1 87,23 0,302 0,896 0.002 

2 12,67 0.120 0,986 0,135 



Standardized Canonical Discriminant Function Coefficients 
(Beta Weights) and Pooled Within-Groups Correlations 
between Discriminating Variables and Canonical 
Discriminant Functions 
(Loadings ) 



Beta Weights Loadings 
FUNCTION 1 FUNCTION 1 



FISCORE 1.005 0.99 

SEX 0.029 -0.15 

AGE 0.02 

LGl -0.01 

LG2 0.05 

TMAWAY -0.00 
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Table 2. Discriminant Function Analysis of Language Strategy Use 



Summary Table of Canonical Discriminant Functions 



PERCENT OF CANONICAL WILKS ' 
FUNCTION VARIANCE CORRELATION LAMBDA SIGNIFICANCE 



1 79.54 0.365 0.833 0.005 

2 20.46 0.195 0.962 0.312 



Standardized Canonical Discriminant Function Coef f ici«ents 
(Beta Weights) Pooled Within-Groups Correlations between 
Discriminating Variables and Canonical Discriminant 
Functions (Loadings) 



Beta Weights Loadings 
FUNCTION 1 FUNCTION 1 



COGAV 1.116 -0.34 

AFFAV -0.709 -0.28 

MEl'^AV 0.23 

SOCAV 0.19 

METAV -0.08 

COMPAV -0.06 
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Table 3. Discriminant Function Analysis of Course Information 



Summary Table of Canonical Discriminant Functions 



PERCENT OF CANONICAL WILKS ' 
FUNCTION VARIANCE CORRELATION LAMBDA SIGNIFICANCE 



1 92.79 0.509 0.721 0.000 

2 7.21 0.163 0.974 0.249 



Standardized Canonical Discriminant Function Coefficients 
(Beta Weights) Pooled Within-Groups Correlations Between 
Discriminating Variables And Canonical Discriminant 
Functions (Loadings) 



Beta Weights Loadings 
FUNCTION 1 FUNCTION 1 

CUMGPA 
SEMGPA 
USEM 
TUCUM 



1.099 
-0.202 
0.104 
0.006 



0.99 
0.78 
0.61 
0.46 



Table 4. Discriminant Function A.nalysis of the Most Important 
Predictors of Group Membership 



Summary Table of Canonical Discriminant Functions 



PERCENT OF CANONICAL WILKS ' 
FUNCTION VARIANCE CORRELATION LAMBDA SIGNIFICANCE 



1 82.09 0.531 0.662 0.000 

2 17.91 0.281 0.922 0.006 



Standardized Canonical Discriminant Function Coefficients 
(Beta V7eights) Pooled Within-Groups Correlations between 
Discriminating Variables and Canonical Discriminant 
Functions (Loadings) 



Beta Weights 

FUNCTION 1 

CUMGPA 0.894 
SEMGPA 

USEM 0.800 
TUCUM 

FISCORE 0.095 
COGAV 

AFFAV 0.341 



Loadings Beta Weights Loadings 

FUNCTION 1 FUNCTION 2 FUNCTION 2 

0.94 0.094 
0.78 

0.59 -0.543 
0.45 



0.915 0.85 
-0.31 

-0.218 0.29 



Group Centroids 

SFD FUNCTION 1 



FUNCTION 2 




Figure 1. Canonical Discriminant Functions of the Host 
Important Predictors of Group Membership Evaluated at 
Group Means 



